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ABSTRACT 

Challenges faced in the evaluation of programs for 
the disabled are discussed, focusing on values orientation, key 
stakeholders, program structure, and measurement issues. Workers in 
the field are presently witnessing a fundamental change in the 
philosophy that underlies programs for persons with disabilities, 
Evaluators must be sensitive to the values orientation behind each 
program-. Disability programs have multiple stakeholders, and an 
evaluator must identify key stakeholders and be responsive to 
consumers and their families. The shift toward community-based 
services is having an effect on program structure, and is increasing 
the number of small programs without the established hierarchical 
structure of larger human services programs. A final consideration is 
the measurement of program outcomes. Acknowledging multiple 
perspectives will require finding or developing multiple measures of 
program outcomes. These are exciting times for programs that serve 
persons with disabilities, but these advances require improved 
evaluation to ensure the best service delivery possible. (Contains 
seven references.) (SLD) 
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The paper I am presenting today came out of my work in developing an 
evaluation resource guide for programs that serve persons with developmental 
disabilities. My consultant for this project was Karen Kirkhart, another presenter 
in this session. As we talked to program staff and examined the literature in this 
area, we became away of contextual issues that pointed to important challenges 
for those of us who evaluate disability programs. We don't mean to imply that 
these challenges are unique - evaluators have certainly encountered many of 
them before - but they are certainly worthy of consideration by those who are 
evaluating disability programs, particularly those in adult services. 

First, a little background in this area. According to the Developmental Disabilities 
Assistance and Bill of Rights Act of 1990, the number of individuals with 
developmental disabilities in the United States is more than three million. People 
with developmental disabilities are defined as those who acquire their disability 
before age 22, so this number fails to include all those who become disabled as 
adults. Programming for persons with disabilities is administered through 
providers in the education, rehabilitation, health and human services sectors, as 
well as from many private non-profit agencies, such as United Way or United 
Cerebral Palsy Associations. 

I have identified four contextual issues that characterize the current climate in disability 
Vn. sen/ices which I would like to discuss today. These issues are: values orientation, 
ti>^ key stakeholders, program structure, and measurement issues. 
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Values Orientation 

We are witnessing a fundamental shift in the philosophy that 
underlies programs for persons with disabilities. In the last century, the focus 
was on custodial care, with vah'e on safety and security within institutional settings. 
About 20 years ago, there was a shift to a focus on skills building and preparing 
people to live and work in sheltered community settings. Evaluations of these 
programs emphasized the extent to which clients were mastering new skills and 
behaviors. In past decade or so, Knoll (1991) pointed to a new paradigm shift . Many 
advocacy groups now argue that an individual with disabilities, no matter how severe 
his impainnents, should be able to be fully included in normalized community living. 

Community integration, empowerment, full inclusion in regular education, and 
quality of life have become the watchwords; however, the best way to achieve these 
goals is still hotly debated among consumers, advocates, clients, service providers, 
and funders. These strongly held values, and debate about how best to meet them, 
have significant implications for evaluation. Certain outcomes will be more highly 
prioritized than others, and some of these outcomes , such as improved quality of life, 
are extremely difficult to measure, as we will discuss presently. 

Programs aimed at transitioning special education students into competitive 
employment in their communities provide one example of the complex values issues 
that can accompany programs for persons with disabilities. Callan (1986) has pointed 
out that because these programs extend into the workplace, they must deal with the 
often unexplicated community corporate and marketplace values. Program directors 
are confronted with value questions such as : 

a) Should greater job preparation efforts be directed at those with the most or 
least severe disabilities? 

b) Can subsidized employment to persons with disabilities be justified when 
non-disabled workers are competing for the same jobs? 

While there are no easy answers to these values questions, Callan argued that part of 
our responsibility as evaluators is to define issues in ways that enable decision- 
makers and other program stakeholders to arrive at more sophisticated, if not less 
troubling, understandings. 

Whatever debate exists about which outcomes are most important and how best 
to meet them will be played out in the evaluation. These differences require 
evaluators to be extremely sensitive to the context in which programs exist and the 
diverse constituencies involved. This brings us to the second issue: Key 
Stakeholders. 
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Key Stakeholders 

While most evaluation situations have multiple stakeholders, disability 
programs really have multiple stakeholders. I was recently appointed to the advisory 
group overseeing the evaluation of the process of closing two state schools in Texas. 
This study was commissioned by the state MHMR, who is the primary stakeholder. 
However. I counted 13 other stakeholder groups: local administrators in the two state 
schools; employees; private providers in the community who will receive some of the 
clients; other state schools who receive other clients; the regional community mental 
retardation authorities who now become responsible for the clients ; service providers 
in other agencies; local government officials whose economy is impacted by the 
closure; the Governor who is still under fire for deciding to close the two state schools; 
state legislators who are being pressured by some constituencies to reverse the 
decision; clients themselves; their families; advocacy groups, some of whom want all 
schools closed and others who want all left open; and last, but not least, the legal 
system, since the closure is part of a court settlement. An evaluator could develop 
whiplash, simply trying to listen to all these diverse groups! 

Clients with multiple handicapping conditions, particularly those acquired early 
in life, have many complex needs. Consequently, programs that work with them often 
find themselves coordinating services with other providers. An adult client who is deaf 
and blind may need employment services, residential services, health services, and 
all these services may be provided by different agencies. This coordination effort in 
itself may call for certain evaluation activities and may also facilitate the sharing of 
information which can be useful to the program's evaluation effort. 

Evaluation has an increasingly critical role to play as persons with 
developmental disabilities move out of institutions into community settings. Dufresne 
(1990) pointed out that many parents oppose deinstitutionalization because they 
believe institutions provide a safer place to monitor the quality of services than do 
decentralized community environments with multiple service sites. Evaluations that 
are responsive to these concerns and demonstrate the safety and quality of community 
services can go a long way toward alleviating family fears during this transitional 
period. 
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One of the most deeply held values in many disability programs is that they 
should be "consumer -driven". This implies a greater role for consumers in program 
evaluation than simply responding to data collection activities. Evaluators of programs 
that serve persons with disabilities will therefore be challenged to involve consumers 
and families in the planning and interpretation of evaluations of their programs. 

Program Structure 

The shift in emphasis to community-based services has had its effects on 
program structure, as well. These programs are often small. Group homes, for 
example, usually serve less than a dozen clients, and many in the disability community 
are calling for smaller, more home-like supported living arrangements, with four 
residents or fewer. Staff in these programs may consist of a couple of administrators 
or program managers and a small direct care staff, although they may be attached to 
larger, parent organizations v/hich provide much of their administrative infrastructure 
and oversight, and are funded largely by huge departments for human resources. In 
many cases, the paid staff is augmented by volunteers, who assist the clients on a 
daily basis. Many of the programs, particularly those involving residential services, are 
administered by private funders. Such programs frequently lack the established 
hierarchical structure of other human service programs, such as those in education; 
when there is only direct service staff and a couple of administrators, there just aren't 
that many layers of bureaucracy. 

This structure has important implications for evaluation. Because there are 
fewer levels of bureaucracy, information may pass more freely through the program's 
informal communication network than it might in a larger organization. In this 
environment, evaluation may provide empirical validation for what staff already 
perceive is happening in the program. On the negative side, fewer resources may be 
available for plann? ^g and carrying out the evaluation. Bradley arid Bersani 
(1990) pointed out that given the pluralism and dispersion in community- 
based services, there won't be enough state employees to handle all 
monitoring and quality assurance . Advocates, case managers, and people 
with disabilities will have to take increasing responsibility for these 
activities. Evaluators will be challenged to include them in the evaluation process - 
not only as targets themselves of evaluation, but also as data collectors and decision- 
makers about what to evaluate. 
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The emerging community integration model promotes individualized outcomes 
and choices and really calls into question how programs are to be defined. Even how 
we label those who receive services has been redefined; many prefer to be called 
"consumers" rather than "patients" or "clients". This paradigm shift raises interesting 
questions for evaluators. If we are moving away from structured programs that deliver 
clearly defined services, to a consumer-driven support system, then evaluation 
approaches dependent on standardized service-delivery models in which all clients 
receive the same intervention simply will not fit. Instead we must develop approaches 
that are responsive to individually determined needs and outcomes. 

Measurement Issues 

Many of the outcomes of interest to programs for persons with disabilities are 
elusive and difficult to measure. For example, quality of life issues are becoming 
increasingly salient. However, quality of life confronts us with a challenging 
measurement problem, since quality of life is hard to define, let alone quantify. 
Definitions of this complex construct vary greatly. Lookiri^ at quality of life issues from 
a health care arena, Caley (1991) identified three different perspectives on quality of 
life. Consumers tend to define it in terms of (a) responsiveness to perceived care 
needs, (b) level of communication, concerns and courtesy, (c) degree of symptom 
relief, and (d) level of functional improvement. By contrast, professionals tend to 
emphasize the degree to which care meets the current technical state of thv>- art and 
freedom to act in the full interest of the patient, while purchasers of services are 
concerned with efficient/appropriate use of health care resources and maximum 
possible contribution of health care to reduction in lost productivity. The point here is 
that perspectives on quality of life vary. While there are many tools that purport to 
measure this construct, the issue becomes whose definition of quality life should be 
used? If we are serious about supports/sen/ices tailored to the needs and desires of 
consumoiS, then we will need to use their definitions - and find or develop measures 
that fit these definitions. 

Feedback from clients about their satisfaction with services is an important 
component of many evaluation systems. Such information is usually collected orally 
or in writing. But programs that serve persons with disabilities face some special 
challenges in collecting information from clients who have limited communication 
skills. Clients who do not read cannot complete most written evaluations, although it 
may be possible to use pictures instead of words. For example, clients can be asked 



6 



to select from a series of faces with expressions depicting various degrees of happy 
and sad affect to indicate how they feel about certain situations. Other clients may be 
unfami' /with the concepts of choice or intrapsychic phenomenon (e.g., self- 
actualization) that are often the focus of our self-report measures. In the past, we have 
often dealt with this problem by interviewing parents or advocates, rather than the 
clients themselves. However, as clients take a more active role as partners in the 
service delivery system, it becomes increasingly important that we find ways to 
communicate directly with them about their needs and preferences. In so doing, we 
must establish that any self-report measures we use with clients have meaning for 
them and are presented in a way to which they can respond. To that end, Biklen and 
Moseley (1988) and Sigelman, et al. (1981) describe carefully constructed interview 
studies with clients who are mentally retarded. 

The outcomes of many human services programs target positive client change. 
We look for increased reading skills in education, or acquisition of jobs by unemployed 
persons, or better utilization of health care services. However, in many programs that 
serve persons with disabilities, particularly adults, the critical goal may be 
maintenance of current functioning, or in the case of clients with degenerative 
Impairments, a decrease in the trajectory of declining functioning. Unlike programs 
that aim to make relatively short-term gains for clients and have them then leave the 
program, many of these clients will be with us a very long time. For example, a 
volunteer advocacy program may have as its goal providing enough support to clients 
to enable them to continue living in community settings. In this situation, the goal isn't 
progress, but maintenance of the ability to live independently in the community. We 
therefore need to recognize that maintenance can be a significant outcome, and 
consequently measures that assess stability, rather than change, must be developed. 
In addition, longitudinal data analysis will become more important to us. 

Conclusions 

In conclusion, these are exciting times for programs that serve persons with 
disabilities. For example, the passage of American with Disabilities Act this year is 
removing many barriers that have kept people with disabilities from participating fully 
in community life. With these opportunities come challenges to the programs that 
serve these individuals to meet their needi. for integration and quality of life more 
effectively. What this means for evaluation is that we are increasingly called on to find 
ways to determine the effects of new and innovative service delivery mechanisms. I 
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would encourage you all to take up the challenges I have touched on today to design 
evaluations that meet the needs of these programs. 
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